{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7e5f9aa4",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/evaluation/prometheus_evaluation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7dc07d6a",
   "metadata": {},
   "source": [
    "# Evaluation using [Prometheus](https://huggingface.co/TheBloke/prometheus-13B-v1.0-GPTQ) model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c142b766",
   "metadata": {},
   "source": [
    "Evaluation is a crucial aspect of iterating over your RAG (Retrieval-Augmented Generation) pipeline. This process has relied heavily on GPT-4. However, a new open-source model named [Prometheus](https://arxiv.org/abs/2310.08491) has recently emerged as an alternative for evaluation purposes.\n",
    "\n",
    "In this notebook, we will demonstrate how you can utilize the Prometheus model for evaluation, integrating it with the LlamaIndex abstractions."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "63e301f5",
   "metadata": {},
   "source": [
    "If you're unfamiliar with the Prometheus model, you might find the paper summary prepared by Andrei informative. It's important to note that this model requires rubric scores to be included in the prompt for effective evaluation. For more detailed information, you can refer to the specific prompts outlined in the notebook."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "eb248085",
   "metadata": {},
   "source": [
    "![Prometheus Paper Card](../data/images/prometheus_paper_card.png)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "36fc6fe6",
   "metadata": {},
   "source": [
    "We will demonstrate the correctness evaluation using the Prometheus model with two datasets from the Llama Datasets. If you haven't yet explored Llama Datasets, I recommend taking some time to read about them [here](https://blog.llamaindex.ai/introducing-llama-datasets-aadb9994ad9e).\n",
    "\n",
    "1. Paul Graham Essay\n",
    "2. Llama2"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "92616d65",
   "metadata": {},
   "source": [
    "### Note: We are showcasing original [Prometheus model](https://huggingface.co/kaist-ai/prometheus-13b-v1.0) for the analysis here. You can re-run the analysis with [quantized version of the model](https://huggingface.co/TheBloke/prometheus-13B-v1.0-GPTQ)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56ae649e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-llms-openai\n",
    "%pip install llama-index-llms-huggingface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2500d100",
   "metadata": {},
   "outputs": [],
   "source": [
    "# attach to the same event-loop\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "17449837",
   "metadata": {},
   "source": [
    "## Download Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e70b44ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.llama_dataset import download_llama_dataset\n",
    "\n",
    "paul_graham_rag_dataset, paul_graham_documents = download_llama_dataset(\n",
    "    \"PaulGrahamEssayDataset\", \"./data/paul_graham\"\n",
    ")\n",
    "\n",
    "llama2_rag_dataset, llama2_documents = download_llama_dataset(\n",
    "    \"Llama2PaperDataset\", \"./data/llama2\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f08c135c",
   "metadata": {},
   "source": [
    "## Define Prometheus LLM hosted on HuggingFace.\n",
    "\n",
    "We hosted the model on HF Inference endpoint using Nvidia A10G GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e165956",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/homebrew/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from llama_index.llms.huggingface import HuggingFaceInferenceAPI\n",
    "\n",
    "HF_TOKEN = \"YOUR HF TOKEN\"\n",
    "HF_ENDPOINT_URL = (\n",
    "    \"https://q3yljc2cypyrvw3i.us-east-1.aws.endpoints.huggingface.cloud\"\n",
    ")\n",
    "\n",
    "prometheus_llm = HuggingFaceInferenceAPI(\n",
    "    model_name=HF_ENDPOINT_URL,\n",
    "    token=HF_TOKEN,\n",
    "    temperature=0.1,\n",
    "    do_sample=True,\n",
    "    top_p=0.95,\n",
    "    top_k=40,\n",
    "    repetition_penalty=1.1,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c0679504",
   "metadata": {},
   "source": [
    "## Prompt templates.\n",
    "\n",
    "We will use same prompts for Prometheus model and GPT-4 to make consistent performance comparision."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e522fbf1",
   "metadata": {},
   "source": [
    "### Correctness Evaluation Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8648503",
   "metadata": {},
   "outputs": [],
   "source": [
    "prometheus_correctness_eval_prompt_template = \"\"\"###Task Description: An instruction (might include an Input inside it), a query, a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given. \n",
    "\t\t\t1. Write a detailed feedback that assesses the quality of the response strictly based on the given score rubric, not evaluating in general. \n",
    "\t\t\t2. After writing a feedback, write a score that is either 1 or 2 or 3 or 4 or 5. You should refer to the score rubric. \n",
    "\t\t\t3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (1 or 2 or 3 or 4 or 5)\" \n",
    "\t\t\t4. Please do not generate any other opening, closing, and explanations. \n",
    "            5. Only evaluate on common things between generated answer and reference answer. Don't evaluate on things which are present in reference answer but not in generated answer.\n",
    "\n",
    "\t\t\t###The instruction to evaluate: Your task is to evaluate the generated answer and reference answer for the query: {query}\n",
    "\t\t\t\n",
    "            ###Generate answer to evaluate: {generated_answer} \n",
    "\n",
    "            ###Reference Answer (Score 5): {reference_answer}\n",
    "            \n",
    "    \t\t###Score Rubrics: \n",
    "            Score 1: If the generated answer is not relevant to the user query and reference answer.\n",
    "            Score 2: If the generated answer is according to reference answer but not relevant to user query.\n",
    "            Score 3: If the generated answer is relevant to the user query and reference answer but contains mistakes.\n",
    "    \t\tScore 4: If the generated answer is relevant to the user query and has the exact same metrics as the reference answer, but it is not as concise.\n",
    "            Score 5: If the generated answer is relevant to the user query and fully correct according to the reference answer.\n",
    "    \n",
    "    \t\t###Feedback:\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7dd8d3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "prometheus_correctness_eval_prompt_template = \"\"\"###Task Description: An instruction (might include an Input inside it), a query, a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given. \n",
    "\t\t\t1. Write a detailed feedback that assesses the quality of the response strictly based on the given score rubric, not evaluating in general. \n",
    "\t\t\t2. After writing a feedback, write a score that is either 1 or 2 or 3 or 4 or 5. You should refer to the score rubric. \n",
    "\t\t\t3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (1 or 2 or 3 or 4 or 5)\" \n",
    "\t\t\t4. Please do not generate any other opening, closing, and explanations. \n",
    "            5. Only evaluate on common things between generated answer and reference answer. Don't evaluate on things which are present in reference answer but not in generated answer.\n",
    "\n",
    "\t\t\t###The instruction to evaluate: Your task is to evaluate the generated answer and reference answer for the query: {query}\n",
    "\t\t\t\n",
    "            ###Generate answer to evaluate: {generated_answer} \n",
    "\n",
    "            ###Reference Answer (Score 5): {reference_answer}\n",
    "            \n",
    "    \t\t###Score Rubrics: \n",
    "            Score 1: If the generated answer is not relevant to the user query and reference answer.\n",
    "            Score 2: If the generated answer is correct according to reference answer but not relevant to user query.\n",
    "            Score 3: If the generated answer is relevant to the user query and correct according to reference answer but has some mistakes in facts.\n",
    "    \t\tScore 4: If the generated answer is relevant to the user query and has the exact same metrics and correct as the reference answer, but it is not as concise.\n",
    "            Score 5: If the generated answer is relevant to the user query and fully correct according to the reference answer.\n",
    "    \n",
    "    \t\t###Feedback:\"\"\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "240e82ad",
   "metadata": {},
   "source": [
    "### Faithfulness Evaluation Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a11597ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "prometheus_faithfulness_eval_prompt_template = \"\"\"###Task Description: An instruction (might include an Input inside it), an information, a context, and a score rubric representing evaluation criteria are given. \n",
    "\t        1. You are provided with evaluation task with the help of information, context information to give result based on score rubrics.\n",
    "            2. Write a detailed feedback based on evaluation task and the given score rubric, not evaluating in general. \n",
    "\t\t\t3. After writing a feedback, write a score that is YES or NO. You should refer to the score rubric. \n",
    "            4. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (YES or NO)” \n",
    "            5. Please do not generate any other opening, closing, and explanations. \n",
    "\n",
    "        ###The instruction to evaluate: Your task is to evaluate if the given piece of information is supported by context.\n",
    "\n",
    "        ###Information: {query_str} \n",
    "\n",
    "        ###Context: {context_str}\n",
    "            \n",
    "        ###Score Rubrics: \n",
    "        Score YES: If the given piece of information is supported by context.\n",
    "        Score NO: If the given piece of information is not supported by context\n",
    "    \n",
    "        ###Feedback: \"\"\"\n",
    "\n",
    "prometheus_faithfulness_refine_prompt_template = \"\"\"###Task Description: An instruction (might include an Input inside it), a information, a context information, an existing answer, and a score rubric representing a evaluation criteria are given. \n",
    "\t\t\t1. You are provided with evaluation task with the help of information, context information and an existing answer.\n",
    "            2. Write a detailed feedback based on evaluation task and the given score rubric, not evaluating in general.\n",
    "\t\t\t3. After writing a feedback, write a score that is YES or NO. You should refer to the score rubric. \n",
    "\t\t\t4. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (YES or NO)\" \n",
    "\t\t\t5. Please do not generate any other opening, closing, and explanations. \n",
    "\n",
    "\t\t\t###The instruction to evaluate: If the information is present in the context and also provided with an existing answer.\n",
    "\n",
    "\t\t\t###Existing answer: {existing_answer} \n",
    "\n",
    "            ###Information: {query_str}\n",
    "\n",
    "            ###Context: {context_msg}\n",
    "            \n",
    "    \t\t###Score Rubrics: \n",
    "            Score YES: If the existing answer is already YES or If the Information is present in the context.\n",
    "            Score NO: If the existing answer is NO and If the Information is not present in the context.\n",
    "    \n",
    "    \t\t###Feedback: \"\"\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "00b50daa",
   "metadata": {},
   "source": [
    "### Relevancy Evaluation Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a47f1341",
   "metadata": {},
   "outputs": [],
   "source": [
    "prometheus_relevancy_eval_prompt_template = \"\"\"###Task Description: An instruction (might include an Input inside it), a query with response, context, and a score rubric representing evaluation criteria are given. \n",
    "            1. You are provided with evaluation task with the help of a query with response and context.\n",
    "            2. Write a detailed feedback based on evaluation task and the given score rubric, not evaluating in general. \n",
    "\t\t\t3. After writing a feedback, write a score that is YES or NO. You should refer to the score rubric. \n",
    "            4. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (YES or NO)” \n",
    "            5. Please do not generate any other opening, closing, and explanations. \n",
    "\n",
    "        ###The instruction to evaluate: Your task is to evaluate if the response for the query is in line with the context information provided.\n",
    "\n",
    "        ###Query and Response: {query_str} \n",
    "\n",
    "        ###Context: {context_str}\n",
    "            \n",
    "        ###Score Rubrics: \n",
    "        Score YES: If the response for the query is in line with the context information provided.\n",
    "        Score NO: If the response for the query is not in line with the context information provided.\n",
    "    \n",
    "        ###Feedback: \"\"\"\n",
    "\n",
    "prometheus_relevancy_refine_prompt_template = \"\"\"###Task Description: An instruction (might include an Input inside it), a query with response, context, an existing answer, and a score rubric representing a evaluation criteria are given. \n",
    "\t\t\t1. You are provided with evaluation task with the help of a query with response and context and an existing answer.\n",
    "            2. Write a detailed feedback based on evaluation task and the given score rubric, not evaluating in general. \n",
    "\t\t\t3. After writing a feedback, write a score that is YES or NO. You should refer to the score rubric. \n",
    "\t\t\t4. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (YES or NO)\" \n",
    "\t\t\t5. Please do not generate any other opening, closing, and explanations. \n",
    "\n",
    "\t\t\t###The instruction to evaluate: Your task is to evaluate if the response for the query is in line with the context information provided.\n",
    "\n",
    "\t\t\t###Query and Response: {query_str} \n",
    "\n",
    "            ###Context: {context_str}\n",
    "            \n",
    "    \t\t###Score Rubrics: \n",
    "            Score YES: If the existing answer is already YES or If the response for the query is in line with the context information provided.\n",
    "            Score NO: If the existing answer is NO and If the response for the query is in line with the context information provided.\n",
    "    \n",
    "    \t\t###Feedback: \"\"\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2fb0de17",
   "metadata": {},
   "source": [
    "Set OpenAI Key for indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2f9b190",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"YOUR OPENAI API KEY\"\n",
    "\n",
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "gpt4_llm = OpenAI(\"gpt-4\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "30f967ea",
   "metadata": {},
   "source": [
    "## Define parser function \n",
    "\n",
    "It will be used in correctness evaluator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15db93bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Tuple\n",
    "import re\n",
    "\n",
    "\n",
    "def parser_function(output_str: str) -> Tuple[float, str]:\n",
    "    # Pattern to match the feedback and response\n",
    "    # This pattern looks for any text ending with '[RESULT]' followed by a number\n",
    "    pattern = r\"(.+?) \\[RESULT\\] (\\d)\"\n",
    "\n",
    "    # Using regex to find all matches\n",
    "    matches = re.findall(pattern, output_str)\n",
    "\n",
    "    # Check if any match is found\n",
    "    if matches:\n",
    "        # Assuming there's only one match in the text, extract feedback and response\n",
    "        feedback, score = matches[0]\n",
    "        score = float(score.strip()) if score is not None else score\n",
    "        return score, feedback.strip()\n",
    "    else:\n",
    "        return None, None"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c72df564",
   "metadata": {},
   "source": [
    "## Define Correctness, FaithFulness, Relevancy Evaluators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea574259",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.evaluation import (\n",
    "    CorrectnessEvaluator,\n",
    "    FaithfulnessEvaluator,\n",
    "    RelevancyEvaluator,\n",
    ")\n",
    "from llama_index.core.callbacks import CallbackManager, TokenCountingHandler\n",
    "import tiktoken\n",
    "\n",
    "\n",
    "# CorrectnessEvaluator with Prometheus model\n",
    "prometheus_correctness_evaluator = CorrectnessEvaluator(\n",
    "    llm=prometheus_llm,\n",
    "    parser_function=parser_function,\n",
    "    eval_template=prometheus_correctness_eval_prompt_template,\n",
    ")\n",
    "\n",
    "# FaithfulnessEvaluator with Prometheus model\n",
    "prometheus_faithfulness_evaluator = FaithfulnessEvaluator(\n",
    "    llm=prometheus_llm,\n",
    "    eval_template=prometheus_faithfulness_eval_prompt_template,\n",
    "    refine_template=prometheus_faithfulness_refine_prompt_template,\n",
    ")\n",
    "\n",
    "# RelevancyEvaluator with Prometheus model\n",
    "prometheus_relevancy_evaluator = RelevancyEvaluator(\n",
    "    llm=prometheus_llm,\n",
    "    eval_template=prometheus_relevancy_eval_prompt_template,\n",
    "    refine_template=prometheus_relevancy_refine_prompt_template,\n",
    ")\n",
    "\n",
    "# Set the encoding model to `gpt-4` for token counting.\n",
    "token_counter = TokenCountingHandler(\n",
    "    tokenizer=tiktoken.encoding_for_model(\"gpt-4\").encode\n",
    ")\n",
    "\n",
    "callback_manager = CallbackManager([token_counter])\n",
    "gpt4_llm.callback_manager = callback_manager\n",
    "\n",
    "# CorrectnessEvaluator with GPT-4 model\n",
    "gpt4_correctness_evaluator = CorrectnessEvaluator(\n",
    "    llm=gpt4_llm,\n",
    "    # parser_function=parser_function,\n",
    ")\n",
    "\n",
    "# FaithfulnessEvaluator with GPT-4 model\n",
    "gpt4_faithfulness_evaluator = FaithfulnessEvaluator(\n",
    "    llm=gpt4_llm,\n",
    "    eval_template=prometheus_faithfulness_eval_prompt_template,\n",
    "    refine_template=prometheus_faithfulness_refine_prompt_template,\n",
    ")\n",
    "\n",
    "# RelevancyEvaluator with GPT-4 model\n",
    "gpt4_relevancy_evaluator = RelevancyEvaluator(\n",
    "    llm=gpt4_llm,\n",
    "    eval_template=prometheus_relevancy_eval_prompt_template,\n",
    "    refine_template=prometheus_relevancy_refine_prompt_template,\n",
    ")\n",
    "\n",
    "# create a dictionary of evaluators\n",
    "prometheus_evaluators = {\n",
    "    \"correctness\": prometheus_correctness_evaluator,\n",
    "    \"faithfulness\": prometheus_faithfulness_evaluator,\n",
    "    \"relevancy\": prometheus_relevancy_evaluator,\n",
    "}\n",
    "\n",
    "gpt4_evaluators = {\n",
    "    \"correctness\": gpt4_correctness_evaluator,\n",
    "    \"faithfulness\": gpt4_faithfulness_evaluator,\n",
    "    \"relevancy\": gpt4_relevancy_evaluator,\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0d14dac0",
   "metadata": {},
   "source": [
    "## Let's create a function to create `query_engine` and `rag_dataset` for different datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7c99c1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.llama_dataset import LabelledRagDataset\n",
    "from llama_index.core import SimpleDirectoryReader, VectorStoreIndex\n",
    "\n",
    "\n",
    "def create_query_engine_rag_dataset(dataset_path):\n",
    "    rag_dataset = LabelledRagDataset.from_json(\n",
    "        f\"{dataset_path}/rag_dataset.json\"\n",
    "    )\n",
    "    documents = SimpleDirectoryReader(\n",
    "        input_dir=f\"{dataset_path}/source_files\"\n",
    "    ).load_data()\n",
    "\n",
    "    index = VectorStoreIndex.from_documents(documents=documents)\n",
    "    query_engine = index.as_query_engine()\n",
    "\n",
    "    return query_engine, rag_dataset"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "03925a73",
   "metadata": {},
   "source": [
    "## Function to run batch evaluations on defined evaluators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a106a98",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.evaluation import BatchEvalRunner\n",
    "\n",
    "\n",
    "async def batch_eval_runner(\n",
    "    evaluators, query_engine, questions, reference=None, num_workers=8\n",
    "):\n",
    "    batch_runner = BatchEvalRunner(\n",
    "        evaluators, workers=num_workers, show_progress=True\n",
    "    )\n",
    "\n",
    "    eval_results = await batch_runner.aevaluate_queries(\n",
    "        query_engine, queries=questions, reference=reference\n",
    "    )\n",
    "\n",
    "    return eval_results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a5a5b50e",
   "metadata": {},
   "source": [
    "## Function to check the distribution of scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c5d39f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from typing import List, Dict\n",
    "\n",
    "\n",
    "def get_scores_distribution(scores: List[float]) -> Dict[str, float]:\n",
    "    # Counting the occurrences of each score\n",
    "    score_counts = Counter(scores)\n",
    "\n",
    "    # Total number of scores\n",
    "    total_scores = len(scores)\n",
    "\n",
    "    # Calculating the percentage distribution\n",
    "    percentage_distribution = {\n",
    "        score: (count / total_scores) * 100\n",
    "        for score, count in score_counts.items()\n",
    "    }\n",
    "\n",
    "    return percentage_distribution"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4c8a631a",
   "metadata": {},
   "source": [
    "## Function to check correctness, faithfulness and relevancy evaluation score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "206f150e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_eval_results(key, eval_results):\n",
    "    results = eval_results[key]\n",
    "    correct = 0\n",
    "    for result in results:\n",
    "        if result.passing:\n",
    "            correct += 1\n",
    "    score = correct / len(results)\n",
    "    print(f\"{key} Score: {round(score, 2)}\")\n",
    "    return score"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "379f5bbd",
   "metadata": {},
   "source": [
    "## Function to compute `Hamming Distance`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f066e8a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hamming_distance(list1, list2):\n",
    "    if len(list1) != len(list2):\n",
    "        raise ValueError(\"Lists must be of the same length\")\n",
    "    return sum(el1 != el2 for el1, el2 in zip(list1, list2))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2300f843",
   "metadata": {},
   "source": [
    "## Evaluation on PaulGraham Essay text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dae4d9b-66d8-4e30-be53-aae3b4d01343",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine, rag_dataset = create_query_engine_rag_dataset(\n",
    "    \"./data/paul_graham\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bec19307",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get questions for evaluation\n",
    "questions = [example.query for example in rag_dataset.examples]\n",
    "\n",
    "# Get reference answers for evaluation\n",
    "reference = [[example.reference_answer] for example in rag_dataset.examples]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "16b9ea6e",
   "metadata": {},
   "source": [
    "### Compute Correctness, Faithfulness and Relevancy Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59d81f10",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 44/44 [00:30<00:00,  1.43it/s]\n",
      "100%|██████████| 132/132 [01:56<00:00,  1.13it/s]\n"
     ]
    }
   ],
   "source": [
    "prometheus_eval_results = await batch_eval_runner(\n",
    "    prometheus_evaluators, query_engine, questions, reference\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57d67e63",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 44/44 [00:26<00:00,  1.66it/s]\n",
      "100%|██████████| 132/132 [02:32<00:00,  1.16s/it]\n"
     ]
    }
   ],
   "source": [
    "gpt4_eval_results = await batch_eval_runner(\n",
    "    gpt4_evaluators, query_engine, questions, reference\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cf51af00",
   "metadata": {},
   "source": [
    "### Correctness Evaluation score distribution with Prometheus Evaluator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c06baac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{3.0: 50.0,\n",
       " 1.0: 43.18181818181818,\n",
       " 5.0: 2.272727272727273,\n",
       " 4.0: 4.545454545454546}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prometheus_scores = [\n",
    "    result.score for result in prometheus_eval_results[\"correctness\"]\n",
    "]\n",
    "get_scores_distribution(prometheus_scores)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e616d8ce",
   "metadata": {},
   "source": [
    "### Correctness Evaluation score distribution with GPT-4 Evaluator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e958e2ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{4.5: 50.0,\n",
       " 5.0: 34.090909090909086,\n",
       " 2.5: 9.090909090909092,\n",
       " 4.0: 2.272727272727273,\n",
       " 3.5: 4.545454545454546}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpt4_scores = [result.score for result in gpt4_eval_results[\"correctness\"]]\n",
    "get_scores_distribution(gpt4_scores)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1b61b75d",
   "metadata": {},
   "source": [
    "### Feedback comparision between prometheus and gpt-4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "116c132e",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = prometheus_eval_results[\"correctness\"][0].query\n",
    "response = prometheus_eval_results[\"correctness\"][0].response\n",
    "reference_answer = reference[0][0]\n",
    "\n",
    "# prometheus feedback and score\n",
    "prometheus_feedback = prometheus_eval_results[\"correctness\"][0].feedback\n",
    "prometheus_score = prometheus_eval_results[\"correctness\"][0].score\n",
    "\n",
    "# GPT4 feedback and score\n",
    "gpt4_feedback = gpt4_eval_results[\"correctness\"][0].feedback\n",
    "gpt4_score = gpt4_eval_results[\"correctness\"][0].score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a44412ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query: In the essay, the author mentions his early experiences with programming. Describe the first computer he used for programming, the language he used, and the challenges he faced. \n",
      "\n",
      "\n",
      "Generated Answer: The author mentions that the first computer he used for programming was the IBM 1401, which was located in the basement of his junior high school. He used an early version of Fortran as the programming language. The author faced challenges in figuring out what to do with the computer, as the only form of input was data stored on punched cards, and he didn't have any. Additionally, he didn't know enough math to do anything interesting with the computer. \n",
      "\n",
      "\n",
      "Reference Answer: The first computer the author used for programming was the IBM 1401, which was used by his school district for data processing. He started using it in 9th grade, around the age of 13 or 14. The programming language he used was an early version of Fortran. The author faced several challenges while using this computer. The only form of input to programs was data stored on punched cards, and he didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but he didn't know enough math to do anything interesting of that type. Therefore, he couldn't figure out what to do with it and in retrospect, he believes there's not much he could have done with it. \n",
      "\n",
      "\n",
      "Prometheus Feedback: The generated response is relevant to the user query and correctly describes the first computer the author used for programming, the programming language he used, and the challenges he faced. However, it has some inaccuracies in the details. The author did not use the IBM 1401 in the basement of his junior high school, but rather in 9th grade, around the age of 13 or 14. The author did not have any data stored on punched cards, but the only form of input was data stored on punched cards. The author did not know enough math to do anything interesting with the computer, but he didn't know enough math to do anything interesting of that type. So the overall score is 3. \n",
      "\n",
      " 3.0 \n",
      "\n",
      "\n",
      "GPT-4 Feedback: The generated answer is highly relevant and almost completely accurate. It correctly identifies the first computer the author used (IBM 1401), the programming language (Fortran), and the challenges he faced (lack of input data and insufficient math knowledge). However, it omits the detail about the author's age and grade level when he started programming, which was included in the reference answer. \n",
      "\n",
      " 4.5\n"
     ]
    }
   ],
   "source": [
    "print(f\"Query: {query} \\n\\n\")\n",
    "print(f\"Generated Answer: {response} \\n\\n\")\n",
    "print(f\"Reference Answer: {reference_answer} \\n\\n\")\n",
    "print(\n",
    "    f\"Prometheus Feedback: {prometheus_feedback} \\n\\n {prometheus_score} \\n\\n\"\n",
    ")\n",
    "print(f\"GPT-4 Feedback: {gpt4_feedback} \\n\\n {gpt4_score}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "efb16b02",
   "metadata": {},
   "source": [
    "#### Observation:\n",
    "\n",
    "The feedback from Prometheus is more detailed, noting that certain specifics were omitted in the generated response, resulting in a score of `3.0`. Conversely, GPT-4's feedback is broader and less specific, awarding a score of `5.0`, despite the absence of some details."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2d7cdab4",
   "metadata": {},
   "source": [
    "### Prometheus Faithfulness and Relevancy Evaluation scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd9fb039",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "faithfulness Score: 0.75\n",
      "relevancy Score: 0.86\n"
     ]
    }
   ],
   "source": [
    "_ = get_eval_results(\"faithfulness\", prometheus_eval_results)\n",
    "\n",
    "_ = get_eval_results(\"relevancy\", prometheus_eval_results)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "026371fa",
   "metadata": {},
   "source": [
    "### GPT-4 Faithfulness and Relevancy Evaluation scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bf36c1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "faithfulness Score: 0.98\n",
      "relevancy Score: 0.95\n"
     ]
    }
   ],
   "source": [
    "_ = get_eval_results(\"faithfulness\", gpt4_eval_results)\n",
    "\n",
    "_ = get_eval_results(\"relevancy\", gpt4_eval_results)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "13154a8b",
   "metadata": {},
   "source": [
    "### Hamming Distance comparison between Prometheus and GPT-4\n",
    "\n",
    "(Lower the better)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b4bbb3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Faithfulness Hamming Distance: 10\n",
      "Relevancy Hamming Distance: 8\n"
     ]
    }
   ],
   "source": [
    "prometheus_faithfulness_scores = [\n",
    "    result.score for result in prometheus_eval_results[\"faithfulness\"]\n",
    "]\n",
    "prometheus_relevancy_scores = [\n",
    "    result.score for result in prometheus_eval_results[\"relevancy\"]\n",
    "]\n",
    "\n",
    "gpt4_faithfulness_scores = [\n",
    "    result.score for result in gpt4_eval_results[\"faithfulness\"]\n",
    "]\n",
    "gpt4_relevancy_scores = [\n",
    "    result.score for result in gpt4_eval_results[\"relevancy\"]\n",
    "]\n",
    "\n",
    "faithfulness_hamming_distance = hamming_distance(\n",
    "    prometheus_faithfulness_scores, gpt4_faithfulness_scores\n",
    ")\n",
    "relevancy_hamming_distance = hamming_distance(\n",
    "    prometheus_relevancy_scores, gpt4_relevancy_scores\n",
    ")\n",
    "\n",
    "print(f\"Faithfulness Hamming Distance: {faithfulness_hamming_distance}\")\n",
    "print(f\"Relevancy Hamming Distance: {relevancy_hamming_distance}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7c73b2fc",
   "metadata": {},
   "source": [
    "#### Observation:\n",
    "\n",
    "The comparison reveals that approximately `77%` and `81%` of the scores are common in case of both `Faithfulness` and `Relevancy` between Prometheus and GPT-4 evaluations respectively. This indicates a decent correlation in terms of faithfulness and relevance scoring between the Prometheus and GPT-4 models."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "04b94dfe",
   "metadata": {},
   "source": [
    "### GPT-4 Cost analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38fda419",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_token_count = token_counter.prompt_llm_token_count\n",
    "completion_token_count = token_counter.completion_llm_token_count\n",
    "\n",
    "total_cost_paul_graham_essay = (\n",
    "    prompt_token_count * 0.03 + completion_token_count * 0.06\n",
    ") / 1000\n",
    "\n",
    "token_counter.reset_counts()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d8349f1a",
   "metadata": {},
   "source": [
    "## Evaluation with Llama2 paper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea3a310a",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine, rag_dataset = create_query_engine_rag_dataset(\"./data/llama2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48cc5c83",
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = [example.query for example in rag_dataset.examples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05131d13",
   "metadata": {},
   "outputs": [],
   "source": [
    "reference = [[example.reference_answer] for example in rag_dataset.examples]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "61c09098",
   "metadata": {},
   "source": [
    "### Compute Correctness, Faithfulness and Relevancy Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "757dec6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [01:02<00:00,  1.61it/s]\n",
      "100%|██████████| 300/300 [04:34<00:00,  1.09it/s]\n"
     ]
    }
   ],
   "source": [
    "prometheus_eval_results = await batch_eval_runner(\n",
    "    prometheus_evaluators, query_engine, questions, reference\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7214c768",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [01:06<00:00,  1.51it/s]\n",
      "100%|██████████| 300/300 [06:22<00:00,  1.27s/it]\n"
     ]
    }
   ],
   "source": [
    "gpt4_eval_results = await batch_eval_runner(\n",
    "    gpt4_evaluators, query_engine, questions, reference\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a842f4e3",
   "metadata": {},
   "source": [
    "### Correctness Evaluation score distribution with Prometheus Evaluator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f4e483c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{3.0: 56.00000000000001, 1.0: 26.0, 5.0: 9.0, 4.0: 8.0, 2.0: 1.0}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prometheus_scores = [\n",
    "    result.score for result in prometheus_eval_results[\"correctness\"]\n",
    "]\n",
    "get_scores_distribution(prometheus_scores)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f26abdfd",
   "metadata": {},
   "source": [
    "### Correctness Evaluation score distribution with GPT-4 Evaluator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dd97ed0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{4.5: 57.99999999999999,\n",
       " 1.0: 6.0,\n",
       " 4.0: 12.0,\n",
       " 5.0: 10.0,\n",
       " 2.0: 5.0,\n",
       " 3.5: 5.0,\n",
       " 2.5: 3.0,\n",
       " 3.0: 1.0}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpt4_scores = [result.score for result in gpt4_eval_results[\"correctness\"]]\n",
    "get_scores_distribution(gpt4_scores)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b957e0d0",
   "metadata": {},
   "source": [
    "### Feedback comparison between prometheus and gpt-4 for correctness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d110a22c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query: Based on the abstract of \"Llama 2: Open Foundation and Fine-Tuned Chat Models,\" what are the two primary objectives achieved in this work, and what is the range of parameters for the large language models developed? \n",
      "\n",
      "\n",
      "Generated Answer: The two primary objectives achieved in this work are the development and release of Llama 2, a collection of pretrained and fine-tuned large language models (LLMs), and the optimization of these models for dialogue use cases. The range of parameters for the large language models developed is from 7 billion to 70 billion. \n",
      "\n",
      "\n",
      "Reference Answer: The two primary objectives achieved in the work described in the abstract of \"Llama 2: Open Foundation and Fine-Tuned Chat Models\" are:\n",
      "\n",
      "1. The development and release of a collection of pretrained and fine-tuned large language models (LLMs) specifically optimized for dialogue use cases.\n",
      "2. The demonstration that these fine-tuned LLMs, referred to as Llama 2-Chat, outperform open-source chat models on most benchmarks tested and may be a suitable substitute for closed-source models, particularly in terms of helpfulness and safety based on human evaluations.\n",
      "\n",
      "The range of parameters for the large language models developed in this work is from 7 billion to 70 billion parameters. \n",
      "\n",
      "\n",
      "Prometheus Feedback: The generated response is relevant to the user query and correctly identifies the two primary objectives of the work described in the abstract of \"Llama  2: Open Foundation and Fine-Tuned Chat Models.\" However, it does not mention the demonstration of the fine-tuned LLMs outperforming open-source chat models on most benchmarks tested, which is a key point in the reference response. The range of parameters for the large language models developed is correctly identified, but the response does not mention the specific models referred to as Llama 2-Chat. So the overall score is 3. \n",
      "\n",
      " 3.0 \n",
      "\n",
      "\n",
      "GPT-4 Feedback: The generated answer is relevant and almost fully correct. It correctly identifies the two primary objectives and the range of parameters for the large language models. However, it misses the detail about Llama 2-Chat outperforming other models on most benchmarks and potentially being a suitable substitute for closed-source models. \n",
      "\n",
      " 4.5\n"
     ]
    }
   ],
   "source": [
    "query = prometheus_eval_results[\"correctness\"][0].query\n",
    "response = prometheus_eval_results[\"correctness\"][0].response\n",
    "reference_answer = reference[0][0]\n",
    "\n",
    "# prometheus feedback and score\n",
    "prometheus_feedback = prometheus_eval_results[\"correctness\"][0].feedback\n",
    "prometheus_score = prometheus_eval_results[\"correctness\"][0].score\n",
    "\n",
    "# GPT4 feedback and score\n",
    "gpt4_feedback = gpt4_eval_results[\"correctness\"][0].feedback\n",
    "gpt4_score = gpt4_eval_results[\"correctness\"][0].score\n",
    "\n",
    "print(f\"Query: {query} \\n\\n\")\n",
    "print(f\"Generated Answer: {response} \\n\\n\")\n",
    "print(f\"Reference Answer: {reference_answer} \\n\\n\")\n",
    "print(\n",
    "    f\"Prometheus Feedback: {prometheus_feedback} \\n\\n {prometheus_score} \\n\\n\"\n",
    ")\n",
    "print(f\"GPT-4 Feedback: {gpt4_feedback} \\n\\n {gpt4_score}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2be4aebd",
   "metadata": {},
   "source": [
    "#### Observation:\n",
    "\n",
    "The feedback from Prometheus is little more precise compared to GPT-4 and it penalises and gives a score of `3.0` but GPT-4 gives a score of `4.5`."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8ea4b274",
   "metadata": {},
   "source": [
    "### Prometheus Faithfulness and Relevancy Evaluation scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53d4e599",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "faithfulness Score: 0.39\n",
      "relevancy Score: 0.57\n"
     ]
    }
   ],
   "source": [
    "_ = get_eval_results(\"faithfulness\", prometheus_eval_results)\n",
    "\n",
    "_ = get_eval_results(\"relevancy\", prometheus_eval_results)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "74c5a9bf",
   "metadata": {},
   "source": [
    "### GPT-4 Faithfulness and Relevancy Evaluation scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3c975d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "faithfulness Score: 0.93\n",
      "relevancy Score: 0.98\n"
     ]
    }
   ],
   "source": [
    "_ = get_eval_results(\"faithfulness\", gpt4_eval_results)\n",
    "\n",
    "_ = get_eval_results(\"relevancy\", gpt4_eval_results)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8c198e09",
   "metadata": {},
   "source": [
    "### Hamming Distance comparison between Prometheus and GPT-4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d9f55cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Faithfulness Hamming Distance: 58\n",
      "Relevancy Hamming Distance: 41\n"
     ]
    }
   ],
   "source": [
    "prometheus_faithfulness_scores = [\n",
    "    result.score for result in prometheus_eval_results[\"faithfulness\"]\n",
    "]\n",
    "prometheus_relevancy_scores = [\n",
    "    result.score for result in prometheus_eval_results[\"relevancy\"]\n",
    "]\n",
    "\n",
    "gpt4_faithfulness_scores = [\n",
    "    result.score for result in gpt4_eval_results[\"faithfulness\"]\n",
    "]\n",
    "gpt4_relevancy_scores = [\n",
    "    result.score for result in gpt4_eval_results[\"relevancy\"]\n",
    "]\n",
    "\n",
    "faithfulness_hamming_distance = hamming_distance(\n",
    "    prometheus_faithfulness_scores, gpt4_faithfulness_scores\n",
    ")\n",
    "relevancy_hamming_distance = hamming_distance(\n",
    "    prometheus_relevancy_scores, gpt4_relevancy_scores\n",
    ")\n",
    "\n",
    "print(f\"Faithfulness Hamming Distance: {faithfulness_hamming_distance}\")\n",
    "print(f\"Relevancy Hamming Distance: {relevancy_hamming_distance}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ab1dc805",
   "metadata": {},
   "source": [
    "#### Observation:\n",
    "\n",
    "The comparison reveals that approximately `44%` of the scores in case of `Faithfulness` and `63%` in case of `Relevancy` are common between Prometheus and GPT-4 evaluations. This indicates a decent amount of correlation in terms of faithfulness and relevance scoring between the Prometheus and GPT-4 models."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "482e92ba",
   "metadata": {},
   "source": [
    "### Feedback comparison between prometheus and gpt-4 for faithfulness and relevancy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0809503d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the query\n",
    "query = questions[0]\n",
    "\n",
    "# Get the response/ generated answer for the query\n",
    "response = prometheus_eval_results[\"faithfulness\"][0].response\n",
    "# Get the retrieved contexts as they are used for faithfulness and relevancy\n",
    "contexts = prometheus_eval_results[\"faithfulness\"][0].contexts\n",
    "\n",
    "# Get the faithfulness and relevancy feedbacks from prometheus model\n",
    "prometheus_faithfulness_feedback = prometheus_eval_results[\"faithfulness\"][\n",
    "    0\n",
    "].feedback\n",
    "prometheus_relevancy_feedback = prometheus_eval_results[\"relevancy\"][\n",
    "    0\n",
    "].feedback\n",
    "\n",
    "# Get the faithfulness and relevancy feedbacks from gpt4 model\n",
    "gpt4_faithfulness_feedback = gpt4_eval_results[\"faithfulness\"][0].feedback\n",
    "gpt4_relevancy_feedback = gpt4_eval_results[\"relevancy\"][0].feedback\n",
    "\n",
    "# Get the failthfulness and relevancy scores from prometheus model\n",
    "prometheus_faithfulness_score = prometheus_eval_results[\"faithfulness\"][\n",
    "    0\n",
    "].score\n",
    "prometheus_relevancy_score = prometheus_eval_results[\"relevancy\"][0].score\n",
    "\n",
    "# Get the faithfulness and relevancy scores from gpt4 model\n",
    "gpt4_faithfulness_score = gpt4_eval_results[\"faithfulness\"][0].score\n",
    "gpt4_relevancy_score = gpt4_eval_results[\"relevancy\"][0].score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9275263f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query: Based on the abstract of \"Llama 2: Open Foundation and Fine-Tuned Chat Models,\" what are the two primary objectives achieved in this work, and what is the range of parameters for the large language models developed? \n",
      "\n",
      "\n",
      "Generated Answer: The two primary objectives achieved in this work are the development and release of Llama 2, a collection of pretrained and fine-tuned large language models (LLMs), and the optimization of these models for dialogue use cases. The range of parameters for the large language models developed is from 7 billion to 70 billion.\n"
     ]
    }
   ],
   "source": [
    "print(f\"Query: {query} \\n\\n\")\n",
    "print(f\"Generated Answer: {response}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19c3d47e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context-1: Llama 2 : Open Foundation and Fine-Tuned Chat Models\n",
      "Hugo Touvron∗Louis Martin†Kevin Stone†\n",
      "Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra\n",
      "Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen\n",
      "Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller\n",
      "Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou\n",
      "Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev\n",
      "Punit Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich\n",
      "Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra\n",
      "Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi\n",
      "Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang\n",
      "Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang\n",
      "Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic\n",
      "Sergey Edunov Thomas Scialom∗\n",
      "GenAI, Meta\n",
      "Abstract\n",
      "In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned\n",
      "large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.\n",
      "Our fine-tuned LLMs, called Llama 2-Chat , are optimized for dialogue use cases. Our\n",
      "models outperform open-source chat models on most benchmarks we tested, and based on\n",
      "ourhumanevaluationsforhelpfulnessandsafety,maybeasuitablesubstituteforclosed-\n",
      "source models. We provide a detailed description of our approach to fine-tuning and safety\n",
      "improvements of Llama 2-Chat in order to enable the community to build on our work and\n",
      "contribute to the responsible development of LLMs.\n",
      "∗Equal contribution, corresponding authors: {tscialom, htouvron}@meta.com\n",
      "†Second author\n",
      "Contributions for all the authors can be found in Section A.1.arXiv:2307.09288v2  [cs.CL]  19 Jul 2023\n"
     ]
    }
   ],
   "source": [
    "print(f\"Context-1: {contexts[0]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd28e3d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context-2: (2021)alsoilluminatesthedifficultiestiedtochatbot-oriented\n",
      "LLMs, with concerns ranging from privacy to misleading expertise claims. Deng et al. (2023) proposes\n",
      "a taxonomic framework to tackle these issues, and Bergman et al. (2022) delves into the balance between\n",
      "potential positive and negative impacts from releasing dialogue models.\n",
      "InvestigationsintoredteamingrevealspecificchallengesintunedLLMs,withstudiesbyGangulietal.(2022)\n",
      "and Zhuoet al. (2023) showcasing a variety ofsuccessful attack typesand their effects onthe generation of\n",
      "harmful content. National security agencies and various researchers, such as (Mialon et al., 2023), have also\n",
      "raisedredflagsaroundadvancedemergentmodelbehaviors,cyberthreats,andpotentialmisuseinareaslike\n",
      "biological warfare. Lastly, broader societal issues like job displacement due to accelerated AI research and an\n",
      "over-reliance on LLMs leading to training data degradation are also pertinent considerations (Acemoglu\n",
      "andRestrepo,2018;AutorandSalomons,2018;Webb,2019;Shumailovetal.,2023). Wearecommittedto\n",
      "continuing our work engaging with the broader policy, academic, and industry community on these issues.\n",
      "7 Conclusion\n",
      "Inthisstudy,wehaveintroduced Llama 2,anewfamilyofpretrainedandfine-tunedmodelswithscales\n",
      "of7billionto70billionparameters. Thesemodelshavedemonstratedtheircompetitivenesswithexisting\n",
      "open-source chat models, as well as competency that is equivalent to some proprietary models on evaluation\n",
      "setsweexamined,althoughtheystilllagbehindothermodelslikeGPT-4. Wemeticulouslyelaboratedonthe\n",
      "methodsandtechniquesappliedinachievingourmodels,withaheavyemphasisontheiralignmentwiththe\n",
      "principlesofhelpfulnessandsafety. Tocontributemoresignificantlytosocietyandfosterthepaceofresearch,\n",
      "wehaveresponsiblyopenedaccessto Llama 2 andLlama 2-Chat . Aspartofourongoingcommitmentto\n",
      "transparency and safety, we plan to make further improvements to Llama 2-Chat in future work.\n",
      "36\n"
     ]
    }
   ],
   "source": [
    "print(f\"Context-2: {contexts[1]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b80a38b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prometheus Faithfulness Feedback: \n",
      "        The information provided in the context is not supported by the given information. The context is about the development and release of Llama 2, a collection of pretrained and fine-tuned large language models (LLMs), and the optimization of these models for dialogue use cases. However, the information provided in the context does not align with the given information. The context does not mention the range of parameters for the large language models developed, which is the primary objective mentioned in the information. The context only talks about the development and release of Llama 2 and its optimization for dialogue use cases, but it does not provide any information about the range of parameters for the large language models developed. So the overall score is NO. [RESULT] NO\n",
      "\n",
      "\n",
      "Prometheus Faithfulness Score: 0.0\n",
      "\n",
      "\n",
      "Prometheus Relevancy Feedback: \n",
      "        The response is not in line with the context information provided. The query asked for the two primary objectives achieved in the work and the range of parameters for the large language models developed. However, the response provided the abstract of the paper and mentioned the authors, which is not relevant to the query. The response also did not mention the two primary objectives achieved in the work or the range of parameters for the large language models developed. So the overall score is NO. [RESULT] NO\n",
      "\n",
      "\n",
      "Prometheus Relevancy Score: 0.0\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"Prometheus Faithfulness Feedback: {prometheus_faithfulness_feedback}\\n\\n\"\n",
    ")\n",
    "print(f\"Prometheus Faithfulness Score: {prometheus_faithfulness_score}\\n\\n\")\n",
    "print(f\"Prometheus Relevancy Feedback: {prometheus_relevancy_feedback}\\n\\n\")\n",
    "print(f\"Prometheus Relevancy Score: {prometheus_relevancy_score}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f0c2eb9e",
   "metadata": {},
   "source": [
    "#### If you compare the feedback and contexts, there is mention of range of parameters in the context and response but the feedback says the model could not find such information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fee1c96e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT-4 Faithfulness Feedback: The given piece of information is well supported by the context. The context clearly states that Llama 2, a collection of pretrained and fine-tuned large language models (LLMs), was developed and released. It also mentions that these models range in scale from 7 billion to 70 billion parameters. Furthermore, the context confirms that these models are optimized for dialogue use cases. Therefore, the information provided is accurate and is corroborated by the context. [RESULT] YES\n",
      "\n",
      "\n",
      "GPT-4 Faithfulness Score: 1.0\n",
      "\n",
      "\n",
      "GPT-4 Relevancy Feedback: The response accurately reflects the context provided. The response correctly identifies the two primary objectives of the work as the development and release of Llama 2, a collection of pretrained and fine-tuned large language models (LLMs), and the optimization of these models for dialogue use cases. This is in line with the information provided in the abstract of the context. The response also correctly states the range of parameters for the large language models developed as being from 7 billion to 70 billion, which is also confirmed in the context. Therefore, the response is in line with the context information provided. [RESULT] YES\n",
      "\n",
      "\n",
      "GPT-4 Relevancy Score: 1.0\n"
     ]
    }
   ],
   "source": [
    "print(f\"GPT-4 Faithfulness Feedback: {gpt4_faithfulness_feedback}\\n\\n\")\n",
    "print(f\"GPT-4 Faithfulness Score: {gpt4_faithfulness_score}\\n\\n\")\n",
    "print(f\"GPT-4 Relevancy Feedback: {gpt4_relevancy_feedback}\\n\\n\")\n",
    "print(f\"GPT-4 Relevancy Score: {gpt4_relevancy_score}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3973f128",
   "metadata": {},
   "source": [
    "#### GPT-4 Evaluates it correctly, unlike prometheus model."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0534ff6c",
   "metadata": {},
   "source": [
    "### GPT-4 Cost analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0feaad6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_token_count = token_counter.prompt_llm_token_count\n",
    "completion_token_count = token_counter.completion_llm_token_count\n",
    "\n",
    "total_cost_llama2 = (\n",
    "    prompt_token_count * 0.03 + completion_token_count * 0.06\n",
    ") / 1000"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2b01ea9b",
   "metadata": {},
   "source": [
    "## Total Cost Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8f63d288",
   "metadata": {},
   "source": [
    "### Prometheus Model - `$2.167` for `144` queries (`44` for Paul Graham Essay and `100` for Llama2 paper) which accounts to `$0.015` per query."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "88a926de",
   "metadata": {},
   "source": [
    "### GPT4 Model - `$22` (total_cost_paul_graham_essay + total_cost_llama2) - which accounts to `$0.15` per query."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dbe28800",
   "metadata": {},
   "source": [
    "## Observation:\n",
    "\n",
    "1. The cost for evaluation (approx.): `$2.167` for Prometheus Model and `$22` for GPT4.\n",
    "2. The Prometheus model, though offering more detailed feedback than GPT-4, occasionally provides incorrect feedback, necessitating cautious application.\n",
    "3. If a generated answer lacks certain facts present in the reference answer, the Prometheus model applies stricter penalties to scores than GPT-4.\n",
    "4. The faithfulness and relevancy feedback of Promethes shows more hallucinations/ wrong interpretations in the feedback compared to GPT-4.\n",
    "5. The commonality between faithfulness and relevancy scores of Promethes and GPT-4 is different across two datasets and so should be used cautiously in production.\n",
    "\n",
    "Note: The endpoint on HF is served on AWS Nvidia A100G · 1x GPU · 80 GB which costs $6.5/h. We used [Prometheus model](https://huggingface.co/kaist-ai/prometheus-13b-v1.0) for the analysis here. We also made similar analysis with [GPTQ Quantized version](https://huggingface.co/TheBloke/prometheus-13B-v1.0-GPTQ) of [Prometheus model](https://huggingface.co/kaist-ai/prometheus-13b-v1.0) and observed abit more hallucinations in feedback compared to original unquantized model. Thanks to authors of the paper and [Tom Jobbins](https://twitter.com/TheBlokeAI) for providing the quantized version of the model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  },
  "vscode": {
   "interpreter": {
    "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
